test: make sure that marked-failing synthetics actually fail by CommanderStorm · Pull Request #1451 · maplibre/maplibre-tile-spec

CommanderStorm · 2026-06-19T02:03:14Z

Currently, when a synthetic test passes we would not know.
This happened to MLT-Wasm, so lets assert that this does not happen.

codecov-commenter · 2026-06-19T02:07:31Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.01%. Comparing base (06cad1f) to head (4db5b9e).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1451      +/-   ##
============================================
+ Coverage     62.66%   69.01%   +6.35%     
- Complexity        0     1003    +1003     
============================================
  Files            94      313     +219     
  Lines         15251    30415   +15164     
  Branches          0     3342    +3342     
============================================
+ Hits           9557    20992   +11435     
- Misses         5694     8296    +2602     
- Partials          0     1127    +1127

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-06-19T02:09:34Z

Performance Comparison

Performance Comparison `main` → `test/remove-unnecessary-test-blockers`

Total Elapsed Time: 2.44s → 2.41s (-1.4%)
CPU Baseline: 82.25µs → 81.64µs (-0.7%)
Benchmark ID: mlt-convert

timing - Function execution time metrics.

+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| Function                             | Calls                        | Avg                            | P95                            | Total                          | % Total                      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::encode                    | 14092 → 14092 (+0.0%)        | 587.13µs → 576.81µs (-1.8%)    | 2.50ms → 2.46ms (-1.6%)        | 8.27s → 8.13s (-1.7%)          | 338.57% → 337.51% (-0.3%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::encode_into               | 28188 → 28188 (+0.0%)        | 191.43µs → 187.95µs (-1.8%)    | 806.91µs → 793.09µs (-1.7%)    | 5.40s → 5.30s (-1.9%)          | 220.81% → 219.98% (-0.4%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_to                     | 28192 → 28191 (-0.0%)        | 141.41µs → 139.63µs (-1.3%)    | 684.03µs → 674.82µs (-1.3%)    | 3.99s → 3.94s (-1.3%)          | 163.14% → 163.44% (+0.2%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| writer::with                         | 2197199 → 2197199 (+0.0%)    | 1.32µs → 1.31µs (-0.8%)        | 1.71µs → 1.71µs (+0.0%)        | 2.91s → 2.89s (-0.7%)          | 119.09% → 119.88% (+0.7%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| mlt::main                            | 1 → 1 (+0.0%)                | 2.44s → 2.41s (-1.2%)          | 2.45s → 2.41s (-1.6%)          | 2.44s → 2.41s (-1.2%)          | 100.00% → 100.00% (+0.0%)    |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| optimizer::analyze                   | 14096 → 14096 (+0.0%)        | 133.25µs → 131.45µs (-1.4%)    | 702.98µs → 685.57µs (-2.5%)    | 1.88s → 1.85s (-1.6%)          | 76.86% → 76.94% (+0.1%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| shared_dict::group_string_properties | 14096 → 14096 (+0.0%)        | 128.45µs → 126.71µs (-1.4%)    | 677.89µs → 668.67µs (-1.4%)    | 1.81s → 1.79s (-1.1%)          | 74.10% → 74.16% (+0.1%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::dict_may_be_beneficial       | 28192 → 28191 (-0.0%)        | 49.65µs → 48.22µs (-2.9%)      | 230.01µs → 226.43µs (-1.6%)    | 1.40s → 1.36s (-2.9%)          | 57.28% → 56.44% (-1.5%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_properties             | 28188 → 28188 (+0.0%)        | 46.35µs → 44.68µs (-3.6%)      | 194.69µs → 189.69µs (-2.6%)    | 1.31s → 1.26s (-3.8%)          | 53.46% → 52.29% (-2.2%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| encode::write_prop                   | 116566 → 116564 (-0.0%)      | 11.02µs → 10.61µs (-3.7%)      | 32.06µs → 31.71µs (-1.1%)      | 1.28s → 1.24s (-3.1%)          | 52.58% → 51.34% (-2.4%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| shared_dict::write_shared_dict       | 24001 → 24000 (-0.0%)        | 30.34µs → 29.18µs (-3.8%)      | 74.37µs → 73.73µs (-0.9%)      | 728.31ms → 700.25ms (-3.9%)    | 29.80% → 29.08% (-2.4%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| tile::from_tile                      | 28192 → 28192 (+0.0%)        | 20.21µs → 19.60µs (-3.0%)      | 88.00µs → 84.54µs (-3.9%)      | 569.81ms → 552.53ms (-3.0%)    | 23.32% → 22.94% (-1.6%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| strings::write_str_col               | 57065 → 57067 (+0.0%)        | 8.29µs → 8.04µs (-3.0%)        | 24.22µs → 24.02µs (-0.8%)      | 472.89ms → 458.94ms (-2.9%)    | 19.35% → 19.06% (-1.5%)      |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| sort::curve_params                   | 14096 → 14096 (+0.0%)        | 16.53µs → 16.49µs (-0.2%)      | 80.25µs → 79.49µs (-0.9%)      | 233.07ms → 232.45ms (-0.3%)    | 9.54% → 9.65% (+1.2%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+
| strings::fsst_try_train              | 40536 → 40536 (+0.0%)        | 4.82µs → 4.72µs (-2.1%)        | 71.00ns → 80.00ns (+12.7%)     | 195.49ms → 191.38ms (-2.1%)    | 8.00% → 7.95% (-0.6%)        |
+--------------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+------------------------------+

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| Function                             | Calls                        | Avg                          | P95                            | Total                          | % Total                    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| tile::from_tile                      | 28192 → 28192 (+0.0%)        | 44.2 KB → 44.2 KB (+0.0%)    | 239.2 KB → 239.2 KB (+0.0%)    | 1.2 GB → 1.2 GB (+0.0%)        | 24.43% → 24.43% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| optimizer::encode                    | 14092 → 14092 (+0.0%)        | 86.2 KB → 86.2 KB (+0.0%)    | 159.2 KB → 159.2 KB (+0.0%)    | 1.2 GB → 1.2 GB (+0.0%)        | 23.84% → 23.84% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| writer::with                         | 2197199 → 2197199 (+0.0%)    | 518 B → 518 B (+0.0%)        | 20.3 KB → 20.3 KB (+0.0%)      | 1.1 GB → 1.1 GB (+0.0%)        | 21.84% → 21.84% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::fsst_try_train              | 40536 → 40536 (+0.0%)        | 16.8 KB → 16.8 KB (+0.0%)    | 3.6 MB → 3.6 MB (+0.0%)        | 664.5 MB → 664.5 MB (+0.0%)    | 13.35% → 13.35% (+0.0%)    |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| shared_dict::group_string_properties | 14096 → 14096 (+0.0%)        | 30.1 KB → 30.1 KB (+0.0%)    | 167.5 KB → 167.5 KB (+0.0%)    | 414.1 MB → 414.1 MB (+0.0%)    | 8.32% → 8.32% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::write_to                     | 28192 → 28191 (-0.0%)        | 7.4 KB → 7.4 KB (+0.0%)      | 35.7 KB → 35.7 KB (+0.0%)      | 204.9 MB → 204.9 MB (+0.0%)    | 4.12% → 4.12% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| shared_dict::write_shared_dict       | 24001 → 24000 (-0.0%)        | 3.4 KB → 3.4 KB (+0.0%)      | 11.5 KB → 11.5 KB (+0.0%)      | 80.2 MB → 80.2 MB (+0.0%)      | 1.61% → 1.61% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::write_str_col               | 57065 → 57067 (+0.0%)        | 893 B → 893 B (+0.0%)        | 3.5 KB → 3.5 KB (+0.0%)        | 48.6 MB → 48.6 MB (+0.0%)      | 0.98% → 0.98% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::build_morton_dict            | 362 → 362 (+0.0%)            | 44.6 KB → 44.6 KB (+0.0%)    | 89.9 KB → 89.9 KB (+0.0%)      | 15.8 MB → 15.8 MB (+0.0%)      | 0.32% → 0.32% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::write_raw_str_data          | 137736 → 137739 (+0.0%)      | 117 B → 117 B (+0.0%)        | 10.8 KB → 10.8 KB (+0.0%)      | 15.4 MB → 15.4 MB (+0.0%)      | 0.31% → 0.31% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| optimizer::analyze                   | 14096 → 14096 (+0.0%)        | 1.0 KB → 1.0 KB (+0.0%)      | 6.5 KB → 6.5 KB (+0.0%)        | 14.0 MB → 14.0 MB (+0.0%)      | 0.28% → 0.28% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::build_hilbert_dict           | 362 → 362 (+0.0%)            | 25.9 KB → 25.9 KB (+0.0%)    | 90.7 KB → 90.7 KB (+0.0%)      | 9.2 MB → 9.2 MB (+0.0%)        | 0.18% → 0.18% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::dict_may_be_beneficial       | 28192 → 28191 (-0.0%)        | 256 B → 256 B (+0.0%)        | 256 B → 256 B (+0.0%)          | 6.9 MB → 6.9 MB (+0.0%)        | 0.14% → 0.14% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::write_prop                   | 116566 → 116564 (-0.0%)      | 41 B → 41 B (+0.0%)          | 400 B → 400 B (+0.0%)          | 4.6 MB → 4.6 MB (+0.0%)        | 0.09% → 0.09% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+
| strings::write_str_plain             | 57068 → 57068 (+0.0%)        | 69 B → 69 B (+0.0%)          | 240 B → 240 B (+0.0%)          | 3.8 MB → 3.8 MB (+0.0%)        | 0.08% → 0.08% (+0.0%)      |
+--------------------------------------+------------------------------+------------------------------+--------------------------------+--------------------------------+----------------------------+

Threads

No threads to compare

Generated with hotpath-rs

Per-tile encoded size diff

zoom	tiles	prev_total	curr_total
0	1	71471	71471
1	4	220990	220990
2	16	615018	615018
3	64	1653881	1653881
4	256	3828995	3828995
5	1024	9716961	9716961
6	4096	32402309	32402309
ALL	5461	48509625	48509625

Top 3 improvements

Top 3 degradations

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

michaelkirk · 2026-06-19T14:20:29Z

I was confused by the PR title, but essentially:

If a known failing synthetic starts newly passing, we want to move it from UNSUPPORTED to supported. Otherwise we might accidentally regress it later.

CommanderStorm · 2026-06-19T22:53:26Z

Yea, reworded it

## 🤖 New release * `mlt-core`: 0.12.0 -> 0.12.1 (✓ API compatible changes) * `mlt`: 0.1.19 -> 0.1.20 * `mlt-py`: 0.1.21 -> 0.1.22 * `mlt-wasm`: 0.1.14 -> 0.1.15 * `mlt-ffi`: 0.1.8 -> 0.1.9 <details><summary>Changelog</summary> ## `mlt-core` <blockquote> ## [0.12.1](rust-mlt-core-v0.12.0...rust-mlt-core-v0.12.1) - 2026-06-20 ### Other - *(deps)* bump fast-mvt from 0.3.2 to 0.4.0 in /rust in the all-cargo-version-updates group ([#1452](#1452)) </blockquote> ## `mlt` <blockquote> ## [0.1.20](rust-mlt-v0.1.19...rust-mlt-v0.1.20) - 2026-06-20 ### Other - update Cargo.lock dependencies </blockquote> ## `mlt-py` <blockquote> ## [0.1.22](python-mlt-v0.1.21...python-mlt-v0.1.22) - 2026-06-20 ### Other - *(deps)* bump fast-mvt from 0.3.2 to 0.4.0 in /rust in the all-cargo-version-updates group ([#1452](#1452)) </blockquote> ## `mlt-wasm` <blockquote> ## [0.1.15](rust-mlt-wasm-v0.1.14...rust-mlt-wasm-v0.1.15) - 2026-06-20 ### Other - make sure that marked-failing synthetics actually fail ([#1451](#1451)) - *(deps)* bump fast-mvt from 0.3.2 to 0.4.0 in /rust in the all-cargo-version-updates group ([#1452](#1452)) </blockquote> ## `mlt-ffi` <blockquote> ## [0.1.9](rust-mlt-ffi-v0.1.8...rust-mlt-ffi-v0.1.9) - 2026-06-20 ### Other - updated the following local packages: mlt-core </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

CommanderStorm force-pushed the test/remove-unnecessary-test-blockers branch from 5ef3787 to c0a023b Compare June 19, 2026 02:07

wip

4db5b9e

CommanderStorm force-pushed the test/remove-unnecessary-test-blockers branch from c0a023b to 4db5b9e Compare June 19, 2026 08:00

CommanderStorm requested a review from louwers June 19, 2026 08:01

CommanderStorm marked this pull request as ready for review June 19, 2026 08:01

Copilot AI review requested due to automatic review settings June 19, 2026 08:01

Copilot AI reviewed Jun 19, 2026

michaelkirk approved these changes Jun 19, 2026

View reviewed changes

CommanderStorm changed the title ~~test: remove unnessary test blockers~~ test: make sure that marked-failing synthetics actually fail Jun 19, 2026

CommanderStorm merged commit a2271b0 into maplibre:main Jun 19, 2026
44 checks passed

CommanderStorm deleted the test/remove-unnecessary-test-blockers branch June 19, 2026 22:53

CommanderStorm mentioned this pull request Jun 19, 2026

chore(rust): release #1453

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: make sure that marked-failing synthetics actually fail#1451

test: make sure that marked-failing synthetics actually fail#1451
CommanderStorm merged 1 commit into
maplibre:mainfrom
CommanderStorm:test/remove-unnecessary-test-blockers

CommanderStorm commented Jun 19, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jun 19, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 19, 2026 •

edited

Loading

Performance Comparison `main` → `test/remove-unnecessary-test-blockers`

timing - Function execution time metrics.

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

Threads

Top 3 improvements

Top 3 degradations

Uh oh!

Copilot AI left a comment

Uh oh!

michaelkirk commented Jun 19, 2026

Uh oh!

Uh oh!

CommanderStorm commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

CommanderStorm commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison main → test/remove-unnecessary-test-blockers

timing - Function execution time metrics.

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

Threads

Top 3 improvements

Top 3 degradations

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

michaelkirk commented Jun 19, 2026

Uh oh!

Uh oh!

CommanderStorm commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CommanderStorm commented Jun 19, 2026 •

edited

Loading

codecov-commenter commented Jun 19, 2026 •

edited

Loading

github-actions Bot commented Jun 19, 2026 •

edited

Loading

Performance Comparison `main` → `test/remove-unnecessary-test-blockers`